Quantitative three-dimensional (Q3D) vision provides numerical information about the actual physical (x, y, z) 3D coordinates of target points in a real world scene. With quantitative 3D vision, a person not only can obtain a three-dimensional perception of a real world scene, but also can obtain numerical information about physical dimensions of objects in the scene and physical distances between objects in the scene. In the past, some Q3D systems have been proposed that use time-of-flight related information or phase information to determine 3D information about a scene. Other Q3D systems have used structured light to determine 3D information about a scene.
The use of time-of-flight information is disclosed in U.S. Pat. No. 6,323,942, entitled, “CMOS-compatible three-dimensional image sensor IC”, which discloses a three-dimensional imaging system that includes a two-dimensional array of pixel light sensing detectors fabricated on a common IC using CMOS fabrication techniques. Each detector has an associated high speed counter that accumulates clock pulses in number directly proportional to time-of-flight (TOF) for a system-emitted pulse to reflect from an object point and be detected by a pixel detector focused upon that point. The TOF data provides a direct digital measure of distance from the particular pixel to a point on the object reflecting the emitted light pulse. In a second embodiment, the counters and high speed clock circuits are eliminated, and instead each pixel detector is provided with a charge accumulator and an electronic shutter. The shutters are opened when a light pulse is emitted and closed thereafter such that each pixel detector accumulates charge as a function of return photon energy falling upon the associated pixel detector. The amount of accumulated charge provides a direct measure of round-trip TOF.
The use of time delay information is disclosed in U.S. Pat. No. 8,262,559, entitled, “Apparatus and method for endoscopic 3D data collection”, which discloses a modulated measuring beam, a light-transmitting mechanism for conducting the measuring beam onto an area to be observed, where the light-transmitting mechanism includes an illuminating lens, in addition to a light-imaging mechanism for imaging a signal beam from the area to be observed at least onto a phase-sensitive image sensor. Time delays, which may correspond to differences in depth in the mm range, result in phase information that makes possible the production of an image that depicts depth and distance information.
The use of structured light to determine physical coordinates of objects in a visual image is disclosed in U.S. Pat. App. Pub. No. 2012/0190923, entitled “Endoscope”; and in C. Schmalz et al., “An endoscopic 3D scanner based on structured light”, Medical Image Analysis, 16 (2012) 1063-1072. A triangulation method is used to measure the topography of a surface. Structured light in the form of projection rays, which may have a range of different color spectra, are incident upon and are reflected from a surface. The reflected rays are observed by a camera that is calibrated to use the reflected color spectra information to determine 3D coordinates of the surface. More specifically, the use of structured light typically involves shining a light pattern on a 3D surface, and determining physical distances based upon a deformation pattern of the light due to contours of the physical object.
An imager array camera has been built that includes a plurality of pixel arrays that can be used to compute scene depth information for pixels in the array. High resolution (HR) images are generated from multiple low resolution (LR) images. A reference viewpoint is selected and an HR image is generated as seen by that viewpoint. A parallax processing technique utilizes the effects of aliasing to determine pixel correspondences for non-reference images with respect to the reference image pixels. Fusion and superresolution are utilized to produce the HR image from the multiple LR images. See, U.S. Pat. No. 8,514,491, entitled “Capturing and Processing Images using Monolithic Camera Array with Heterogeneous Imager”; U.S. Pat. App. Pub. No. 2013/0070060, entitled, Systems and Methods for Determining Depth from multiple Views of a Scene that Include Aliasing using Hypothesized Fusion”; and K. Venkataraman et al., PiCam: An ultra-Thin high Performance Monolithic Camera Array.
FIG. 1 is an illustrative drawing showing details of a known imager sensor 180 in accordance with some embodiments. The image sensor 180 includes an optic array of lens stacks 182 and an array of sensors 184. Each sensor in the array includes a two dimensional arrangement of pixels having at least two pixels in each dimension. Each sensor includes a lens stack 186 that creates a separate optical channel that resolves an image onto a corresponding arrangement of pixels disposed in a focal 188 plane of the lens stack. The pixels act as light sensors and each focal plane 188 with its multiple pixels acts as an image sensor. Each sensor with its focal plane 182 occupies a region of the sensor array different from regions of the sensor array occupied by other sensors and focal planes.
FIG. 2 is an illustrative drawing showing a simplified plan view of the known imager sensor array 180 of FIG. 1 that includes sensors S11 through S33. The imager sensor array 180 is fabricated on a semiconductor chip to include a plurality of sensors S11 through S33 Each of the sensors S11 through S33 includes a plurality of pixels (e.g., 0.32 Mega pixels) and is coupled to peripheral circuitry (not shown) that includes independent read-out control and pixel digitization. In some embodiments, the sensors S11 through S33 are arranged into a grid format as illustrated in FIG. 2. In other embodiments, the sensors are arranged in a non-grid format. For example, the imagers may be arranged in a circular pattern, zigzagged pattern or scattered pattern or an irregular pattern including sub-pixel offsets.
Each sensor pixel includes a microlens pixel stack. FIG. 3 is an illustrative drawing of a known microlens pixel stack of the sensors of FIGS. 1-2. The pixel stack 800 includes a microlens 802, which sits atop an oxide layer 804, which is typically beneath the oxide layer 804 there may be a color filter 806, which is disposed above a nitride layer 808, which is disposed above a second oxide layer 810, which and sits atop a silicon 812 that layer includes the active area 814 of the sensor (typically a photodiode). The primary role of a microlens 802 is to gather the light incident on its surface and to focus that light onto the small active area 814. The pixel aperture 816 is determined by the spread of the microlens, which collects the light and focuses it on the active area 814.
Additional information concerning the above-described known imager sensor array architecture is provided in U.S. Pat. No. 8,514,491, issued, Aug. 20, 2013; and in U.S. Patent Application Pub. No. 20013/0070060, published Mar. 21, 2013.